Comparing Predictive Accuracy of Real Estate Pricing Models: An Applied Study for the City of Montreal.

Rapport de recherche

Recurrent events counted in evaluations of predictive accuracy

Objectives: To describe an alternate approach for the calculation of sensitivity and specificity when analyzing the accuracy of screening tools, which can be used when standard calculations may be inappropriate. Sensitivity^ER (ER denoting event rate) is the number of events correctly predicted, divided by the total number of events. Specificity^ER is the amount of time that study participants are predicted to be event negative, divided by the total amount of participant observed time. Variance estimates for these statistics are constructed by bootstrap resampling, taking into account event dependence.

Methods: Standard and alternate approaches for calculating sensitivity and specificity were applied to hospital falls risk screening tool data. In this application, the outcome of interest was a recurrent event, there were multiple applications of the screening tool, delays in screening tool  completion, and patients' follow-up durations were unequal.

Results: Application of sensitivity^ER and specificity^ER to this data not only provided a clearer description of the screening tool's overall accuracy, but also allowed examination of accuracy over time, accuracy in predicting specific event numbers, and evaluation of the added value that screening tool reapplications may have.

Conclusion: Sensitivity^ER and specificity^ER provide a valuable approach to screening tool evaluation in the clinical setting.

Design-related bias in hospital fall risk screening tool predictive accuracy evaluations: systematic review and meta-analysis.

Introduction: Fall risk screening tools are frequently used as a part of falls prevention programs in hospitals. Design-related bias in evaluations of tool predictive accuracy could lead to overoptimistic results, which would then contribute to program failure in practice.

Methods: A systematic review was undertaken. Two blind reviewers assessed the methodology of relevant publications into a four-point classification system adapted from multiple sources. The association between study design classification and reported results was examined using linear regression with clustering based on screening tool and robust variance estimates with point estimates of Youden Index (= sensitivity + specificity - 1) as the dependent variable. Meta-analysis was then performed pooling data from prospective studies.

Results: Thirty-five publications met inclusion criteria, containing 51 evaluations of fall risk screening tools. Twenty evaluations were classified as retrospective validation evaluations, 11 as prospective (temporal) validation evaluations, and 20 as prospective (external) validation evaluations. Retrospective evaluations had significantly higher Youden Indices (point estimate [95% confidence interval]: 0.22 [0.11, 0.33]). Pooled Youden Indices from prospective evaluations demonstrated the STRATIFY, Morse Falls Scale, and nursing staff clinical judgment to have comparable accuracy.

Discussion: Practitioners should exercise caution in comparing validity of fall risk assessment tools where the evaluation has been limited to retrospective classifications of methodology. Heterogeneity between studies indicates that the Morse Falls Scale and STRATIFY may still be useful in particular settings, but that widespread adoption of either is unlikely to generate benefits significantly greater than that of nursing staff clinical judgment.

Enhancing predictive accuracy of cardiac autonomic neuropathy using blood biochemistry features and iterative multitier ensembles

Blood biochemistry attributes form an important class of tests, routinely collected several times per year for many patients with diabetes. The objective of this study is to investigate the role of blood biochemistry for improving the predictive accuracy of the diagnosis of cardiac autonomic neuropathy (CAN) progression. Blood biochemistry contributes to CAN, and so it is a causative factor that can provide additional power for the diagnosis of CAN especially in the absence of a complete set of Ewing tests. We introduce automated iterative multitier ensembles (AIME) and investigate their performance in comparison to base classifiers and standard ensemble classifiers for blood biochemistry attributes. AIME incorporate diverse ensembles into several tiers simultaneously and combine them into one automatically generated integrated system so that one ensemble acts as an integral part of another ensemble. We carried out extensive experimental analysis using large datasets from the diabetes screening research initiative (DiScRi) project. The results of our experiments show that several blood biochemistry attributes can be used to supplement the Ewing battery for the detection of CAN in situations where one or more of the Ewing tests cannot be completed because of the individual difficulties faced by each patient in performing the tests. The results show that AIME provide higher accuracy as a multitier CAN classification paradigm. The best predictive accuracy of 99.57% has been obtained by the AIME combining decorate on top tier with bagging on middle tier based on random forest. Practitioners can use these findings to increase the accuracy of CAN diagnosis.

Complex Symbolic Sequence Encodings for Predictive Monitoring of Business Processes

This paper addresses the problem of predicting the outcome of an ongoing case of a business process based on event logs. In this setting, the outcome of a case may refer for example to the achievement of a performance objective or the fulfillment of a compliance rule upon completion of the case. Given a log consisting of traces of completed cases, given a trace of an ongoing case, and given two or more possible out- comes (e.g., a positive and a negative outcome), the paper addresses the problem of determining the most likely outcome for the case in question. Previous approaches to this problem are largely based on simple symbolic sequence classification, meaning that they extract features from traces seen as sequences of event labels, and use these features to construct a classifier for runtime prediction. In doing so, these approaches ignore the data payload associated to each event. This paper approaches the problem from a different angle by treating traces as complex symbolic sequences, that is, sequences of events each carrying a data payload. In this context, the paper outlines different feature encodings of complex symbolic sequences and compares their predictive accuracy on real-life business process event logs.

(Semi-)Predictive Discretization During Model Selection

In this paper, we present an approach to discretizing multivariate continuous data while learning the structure of a graphical model. We derive the joint scoring function from the principle of predictive accuracy, which inherently ensures the optimal trade-off between goodness of fit and model complexity (including the number of discretization levels). Using the so-called finest grid implied by the data, our scoring function depends only on the number of data points in the various discretization levels. Not only can it be computed efficiently, but it is also independent of the metric used in the continuous space. Our experiments with gene expression data show that discretization plays a crucial role regarding the resulting network structure.

Modeling Linkage Disequilibrium Increases Accuracy of Polygenic Risk Scores

Polygenic risk scores have shown great promise in predicting complex disease risk and will become more accurate as training sample sizes increase. The standard approach for calculating risk scores involves linkage disequilibrium (LD)-based marker pruning and applying a p value threshold to association statistics, but this discards information and can reduce predictive accuracy. We introduce LDpred, a method that infers the posterior mean effect size of each marker by using a prior on effect sizes and LD information from an external reference panel. Theory and simulations show that LDpred outperforms the approach of pruning followed by thresholding, particularly at large sample sizes. Accordingly, predicted R(2) increased from 20.1% to 25.3% in a large schizophrenia dataset and from 9.8% to 12.0% in a large multiple sclerosis dataset. A similar relative improvement in accuracy was observed for three additional large disease datasets and for non-European schizophrenia samples. The advantage of LDpred over existing methods will grow as sample sizes increase.

Biomedical engineering for healthy ageing. Predictive tools for falls

Falls are common and burdensome accidents among the elderly. About one third of the population aged 65 years or more experience at least one fall each year. Fall risk assessment is believed to be beneficial for fall prevention. This thesis is about prognostic tools for falls for community-dwelling older adults. We provide an overview of the state of the art. We then take different approaches: we propose a theoretical probabilistic model to investigate some properties of prognostic tools for falls; we present a tool whose parameters were derived from data of the literature; we train and test a data-driven prognostic tool. Finally, we present some preliminary results on prediction of falls through features extracted from wearable inertial sensors. Heterogeneity in validation results are expected from theoretical considerations and are observed from empirical data. Differences in studies design hinder comparability and collaborative research. According to the multifactorial etiology of falls, assessment on multiple risk factors is needed in order to achieve good predictive accuracy.

Quantifying and Comparing the Accuracy of Binary Biomarkers When Predicting a Failure Time Outcome

The positive and negative predictive value are standard measures used to quantify the predictive accuracy of binary biomarkers when the outcome being predicted is also binary. When the biomarkers are instead being used to predict a failure time outcome, there is no standard way of quantifying predictive accuracy. We propose a natural extension of the traditional predictive values to accommodate censored survival data. We discuss not only quantifying predictive accuracy using these extended predictive values, but also rigorously comparing the accuracy of two biomarkers in terms of their predictive values. Using a marginal regression framework, we describe how to estimate differences in predictive accuracy and how to test whether the observed difference is statistically significant.

Accuracy of in vivo coronary plaque morphology assessment: a validation study of in vivo virtual histology compared with in vitro histopathology

OBJECTIVES: The goal of the present study was to compare the accuracy of in vivo tissue characterization obtained by intravascular ultrasound (IVUS) radiofrequency (RF) data analysis, known as Virtual Histology (VH), to the in vitro histopathology of coronary atherosclerotic plaques obtained by directional coronary atherectomy. BACKGROUND: Vulnerable plaque leading to acute coronary syndrome (ACS) has been associated with specific plaque composition, and its characterization is an important clinical focus. METHODS: Virtual histology IVUS images were performed before and after a single debulking cut using directional coronary atherectomy. Debulking region of in vivo histology image was predicted by comparing pre- and post-debulking VH images. Analysis of VH images with the corresponding tissue cross section was performed. RESULTS: Fifteen stable angina pectoris (AP) and 15 ACS patients were enrolled. The results of IVUS RF data analysis correlated well with histopathologic examination (predictive accuracy from all patients data: 87.1% for fibrous, 87.1% for fibro-fatty, 88.3% for necrotic core, and 96.5% for dense calcium regions, respectively). In addition, the frequency of necrotic core was significantly higher in the ACS group than in the stable AP group (in vitro histopathology: 22.6% vs. 12.6%, p = 0.02; in vivo virtual histology: 24.5% vs. 10.4%, p = 0.002). CONCLUSIONS: Correlation of in vivo IVUS RF data analysis with histopathology shows a high accuracy. In vivo IVUS RF data analysis is a useful modality for the classification of different types of coronary components, and may play an important role in the detection of vulnerable plaque.